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HPRef.No. 100110275 

HIGH RESOLUTION IMAGE RECONSTRUCTION 

FIELD OF THF INVENTION 



[0001 J The present invention relates generally to digital image processing and 
particularly to interpolation of a mixed resolution image sequence for high-resolution image 



more 



reconstruction. 



BACKGROUND O F THE INVENTION 

[0002] Digital video processing refers to manipulation of the digital video 
bitstream. Some applications of digital video processing include standards conversion, image 
enhancement, high resolution image reconstruction, etc. 

[0003] One known method for high-resolution image reconstruction includes the 
use of spatial interpolation to produce a low spatial frequency image estimate from a captured 
low-resolution image frame. The estimate, however, may appear blurry as it does not 
incorporate any related high spatial frequency information from neighboring high-resolution 
image frames and, consequently, does not allow large size prints of the captured low-resolution 
image. 

[0004] Another known high-resolution image reconstruction method uses non- 
linear image interpolation which improves somewhat the spatial resolution of the resulting 
estimate, but the improvement is based on statistical information about high spatial frequency 
content. Furthermore, this method requires training with a set of parameters (images). 

[0005] None of the known image reconstruction methods utilize high spatial 
frequency data from closely related high-resolution image frames. 

SUMMARY OF THE INVENTION 

[0006] The present disclosure is generally directed to a high resolution image 
reconstruction method, comprising generating at least one image sequence of temporally 
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related high and low resolution image frames, each of the high resolution image frames 
including a low spatial frequency component and a high spatial frequency component; using 
spatial interpolation to generate a low spatial frequency component from a low resolution 
image frame of the image sequence; generating a high spatial frequency component from at 
least one high resolution image frame of the image sequence, the high resolution image frame 
being closely related to the low resolution image frame; remapping the high spatial frequency 
component to a motion-compensated high spatial frequency component estimate of the low 
resolution image frame; and adding the motion-compensated high spatial frequency component 
estimate of the low resolution image frame to the generated low spatial frequency component 
of the low resolution image frame to form a reconstructed high resolution image of the low 
resolution image frame. 

[0007] These and other aspects of the present invention will become apparent from 
a review of the accompanying drawings and the following detailed description of the preferred 
embodiments of the present invention. 



BRIEF DESCRIPTION OF THE DRAWINGS 

[0008] The invention is generally shown by way of example in the accompanying 
drawings in which: 

Figure 1 is a schematic representation of an embodiment of an image sensor having 
integrated light sensing and subsequent signal processing capability for use in accordance with 
the present invention; 

Figure 2 is a schematic representation of an embodiment of an image downsampling 
module for use in accordance with the present invention; 

Figure 3 is a schematic representation of an embodiment of a mixed spatial resolution 
image sequence in accordance with the present invention; 

Figure 4 is a schematic representation of an embodiment of an image downsampling 
(decimation) in accordance with the present invention; 
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Figure 5 is a schematic representation of an embodiment of a subpixel motion 
processing in accordance with the present invention; 

Figure 6 is a schematic representation of an embodiment of a first spatial interpolation 
in accordance with the present invention; 

Figure 7 is a schematic representation of an embodiment of high spatial frequency 
image component determination in accordance with the present invention; 

Figure 8 is a schematic representation of an embodiment of a second spatial 
interpolation in accordance with the present invention; 

Figure 9 is a schematic representation of an embodiment of motion-compensated image 
remapping in accordance with the present invention; 

Figure 10 is a schematic representation of an embodiment of motion confidence scalar 
field processing to form per-pixel weights a in accordance with the present invention; 

Figure 11 is a schematic representation of an embodiment of high spatial frequency 
image component estimation using the per-pixel weights a of Figure 10 in accordance with the 
present invention; and 

Figure 12 is a schematic representation of an embodiment of high-resolution image 
reconstruction in accordance with the present invention. 

DETAILED DESCRIPTIO N OF THE PREFERRED EMBODIMENTS 

[0009] Hereinafter, some preferred embodiments of the present invention will be 
described in detail with reference to the related drawings of Figures 1 - 12. Additional 
embodiments, features and/or advantages of the invention will become apparent from the 
ensuing description or may be learned by practicing the invention. 

[0010] The drawings are not to scale with like numerals referring to like features 
throughout both the drawings and the description. 



25306545.3 (6051-144) 



4 



[0011] The following description includes the best mode presently contemplated 
for carrying out the invention. This description is not to be taken in a limiting sense, but is 
made merely for the purpose of describing the general principles of the invention. 

[0012] Current CMOS (complementary metal-oxide semiconductor) image sensor 
technology allows the integration of light sensing elements and subsequent digital image signal 
processing. One known example of such CMOS technology allows a digital camera to switch 
seamlessly between high resolution and low resolution video without sacrificing the image 
quality of either. Specifically, signals from adjacent pixels can be combined into groups and 
read as one larger pixel with the size and configuration of a pixel group being variable and 
controlled by specialized integrated circuitry. Smaller pixels offer the benefit of higher 
resolution and sharper images. Larger pixels offer the benefit of higher light sensitivity 
required for low light still scenes and video applications. In other words, a CMOS image 
sensor of this type can be addressed in variable resolutions and thus, is capable of capturing a 
high-resolution video frame in the midst of recording motion video since pixel sizing can be 
done instantaneously. 

[0013] Turning to Figure 1, a variable resolution CMOS image sensor, generally 
referred to by a reference numeral 20, preferably comprises an image sensor array 22 
operatively coupled to an image downsampling module 24. Image sensor array 22 is adapted to 
sense light from a scene, as generally shown in Fig. 1. Image downsampling module 24 is 
operatively coupled between a downsampling controller 26 and RAM (random access 
memory) 28 (Fig. 1), i.e., the size of output images may be controlled for each frame. For 
example, RAM input signal 30 (Fig. 1) may be a sequence of image frames, taken 1/30 
seconds apart, having different spatial resolutions such as low (640 x 480 pixels) spatial 
resolution image frames and high (1920 x 1440 pixels) spatial resolution image frames. A 
mixed spatial resolution image sequence of this type is shown schematically in Fig. 3 and 
generally referred to by a reference numeral 32. For example, image sequence 32 (Fig. 3) may 
comprise high spatial resolution image frame A<, followed by a series of low spatial resolution 
image frames a, - a 3 which is followed by high spatial resolution image frame A, which, in 
turn, is followed by a series of low spatial resolution image frames a 5 - a 7 , which is followed 
by high spatial resolution image frame A«, etc. 

[0014] Figure 2 schematically illustrates an embodiment of the internal components 
of image downsampling module 24, namely, a low pass anti-aliasing filter 34 and a spatial 
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subsampler 36. Low pass anti-aliasing filter 34 is operatively coupled between image sensor 
array 22 and spatial subsampler 36. Spatial subsampler 36 outputs to RAM 28, as generally 
depicted in Fig. 2. Image downsampling is performed by filtering and subsampling the input 
image data. Those skilled in the art will note that the two steps may be combined for efficient 
implementation. According to this embodiment, the image data is convolved with a low pass 
anti-aliasing filtering 34. Subsampling is controlled via downsampling controller 26. For 
example, downsampling controller 26 may instruct spatial subsampler 36 to discard 2 of 3 
samples, in each dimension, to downsample a 1920 x 1440 pixel frame to a 640 x 480 pixel 
frame. 

[0015] Due to practical limitations in memory and bandwidth, not all captured 
image frames may be of high spatial resolution, i.e., only some high spatial resolution image 
frames such as, for example, Ac, A* As, are retained with the rest being automatically 
downsampled (by CMOS image sensor 20) such as, for example, a, - a 3 , a 5 - a 7 , as generally 
shown in Fig. 3. High spatial resolution images such as A* A,, As will produce good quality 
large size prints, i.e. may be printed directly without further processing. However, low spatial 
resolution images such as a, - a 3 , a s - a 7 will not produce good quality large size prints 
because of low spatial resolution. On the other hand, the upper limit in size of a good quality 
print is determined, to a great extent, by the spatial resolution of the input image sequence. 
Therefore, being capable of reconstructing a good quality high spatial resolution image from an 
available low-resolution image is a desirable feature for digital cameras and/or image 
processing software. 

[0016] In order to produce a good quality large size print from a low spatial 
resolution image frame such as, for example, a,, a fairly good high spatial resolution image 
estimate A, must be formed of the original (unavailable) image frame A,. In general, A, is a 
combination of high and low spatial frequency components, A,„, A 1L , respectively, i.e. A, = 
A 1L + A 1H . The low spatial frequency component A 1L of original (unavailable) A, may be 
generated via spatial interpolation, as generally illustrated in Fig. 8. Specifically, available low 
resolution image a, (Fig. 3) may be passed through a spatial upsampler 38 (Fig. 8) and a low 
pass reconstruction filter 40 (Fig. 8) to produce low spatial frequency component A 1L which is 
essentially a "blurry" version of unavailable high resolution image A,. Spatial upsampler 38 
may be programmed to utilize standard bicubic or bilinear algorithms. Alternatively, spatial 
upsampler 38 may be programmed to utilize least squares error minimization algorithms to 
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produce a relatively sharper upsampled image in accordance with a preferred embodiment of 
the present invention. In particular, given a downsampling matrix B, applying its pseudo- 
inverse B + to the image calculates an upsampled approximation that is closest, in the least 
squares sense, to the original image. 

[0017] Figure 4 schematically illustrates an embodiment of image downsampling 
(decimation) in accordance with a preferred embodiment of the present invention. Specifically, 
available and closely related high spatial resolution image A* may be passed through low pass 
anti-aliasing filter 34 and spatial subsampler 36 (of image downsampling module 24 to 
produce a low spatial resolution image a 0 , i.e. a downsampled version of itself. As generally 
depicted in Fig. 5, the generated low spatial resolution image ao (Fig. 4) and the available low 
spatial resolution image a, (Fig. 3) may be fed into a subpixel motion processor 42 to generate 
subpixel accuracy motion vector field MVF 0I and associated motion confidence scalar field 
CSF 01 , respectively, i.e. each motion vector would have an associated confidence scalar value. 
Furthermore, the generated low spatial resolution image ao (Fig. 4) may be fed into spatial 
upsampler 38 and low pass reconstruction filter 40, as generally illustrated in Fig. 6, to produce 
low spatial frequency component Ao L which is essentially a "blurry" version of originally 
available high spatial resolution image A« (Fig. 3). The generated low spatial frequency 
component Aol may then be subtracted from originally available high spatial resolution image 
A„ (Fig. 3) to obtain a residual high spatial frequency component A™, as generally shown in 
Fig. 7. 

[0018] In accordance with another embodiment of the present invention, the 
obtained residual high spatial frequency component Ao H and subpixel accuracy motion vector 
field MVFo, may be fed into a remap module 44 to produce an unweighted motion- 
compensated (remapped) residual high spatial frequency component A' 1H , as generally 
depicted in Fig. 9. Moreover, the generated motion confidence scalar field CSF 0 , may be 
processed via a standard weight function generator 46 to form corresponding per-pixel weights 
a, as generally shown in Fig. 10. Weight function generator 46 preferably includes a pixel 
weight look-up table (not shown). The weights a basically control how much of the remapped 
residual high spatial frequency component A',„ is added to the low spatial frequency 
component A 1L to form the final reconstructed high spatial resolution image estimate A,. 
Specifically, the weights a are multiplied by the unweighted remapped residual high spatial 
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frequency component A' IH to generate corresponding weighted remapped residual high spatial 
frequency component A,„, as generally shown in Fig. 11. The weighted residual high spatial 
frequency component A 1H and low spatial frequency component A 1L may then be added to 
obtain a fairly good high spatial resolution image estimate A, of original, unavailable high 
spatial resolution image A,. In this regard, one possible method for computing motion 
confidence is to apply cross-correlation matching to determine when the motion is well 
described by a simple translation. For motion vectors with full confidence (a = 1) the full high 
spatial frequency residuals may be added. For motion vectors with low confidence, less of the 
high spatial frequency residuals may be added to control for possible ghosting artifacts. The 
degree of motion estimation accuracy essentially controls the sharpness of the reconstructed 
high-resolution image. 

[0019] A person skilled in the art would readily recognize that the above-described 
novel high resolution image reconstruction method may be easily generalized to take 
advantage of additional high-resolution frames that might be available in the mixed pipeline 
output. A person skilled in the art should also recognize that given a low resolution image, the 
described image reconstruction method may be expanded to obtain a good high resolution 
image estimate not just from one neighboring (temporally related) high resolution image, but 
also from two or more closely related high resolution images. The novel high-resolution image 
reconstruction method may be used in a variety of applications such as to restore a video 
sequence to a higher resolution, allowing for high quality digital zoom, and/or to improve 
printing from video frames. 

[0020] Furthermore, the above-described components of Figs. 5 - 12, inclusive, 
may be integrated in hardware or software form depending on desired application. 

[0021] Other components and/or configurations may be utilized in the above- 
described embodiments, provided such other components and/or configurations do not depart 
from the intended purpose and scope of the present invention. While the present invention has 
been described in detail with regards to one or more preferred embodiments, it should also be 
appreciated that various modifications and variations may be made in the present invention 
without departing from the scope or spirit of the invention. In this regard it is important to note 
that practicing the invention is not limited to the applications described hereinabove. Other 
applications and/or alterations may become apparent to those skilled in the art. 
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[0022] It should be appreciated by a person skilled in the art that features illustrated 
or described as part of one embodiment may also be used in other embodiments. It is, 
therefore, intended that the present invention cover all such modifications, embodiments and 
variations as long as such modifications, embodiments and variations remain within the scope 
of the appended claims and their equivalents. 
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